Subject determination method, computer program product for determining subject, and camera

ABSTRACT

A photographic subject determination method includes: a binarization step of creating a plurality of binarized images of a subject image, based upon color information or luminance information in the subject image; an evaluation value calculation step of, for each of the plurality of binarized images, calculating an evaluation value that is used for specifying at least one of a position, a size, and a shape of a photographic subject within the subject image; and a photographic subject specification step of specifying at least one of the position, the size, and the shape of a photographic subject within the subject image, based upon the evaluation value.

INCORPORATION BY REFERENCE

The disclosures of the following priority applications are hereinincorporated by reference: Japanese Patent Application No. 2010-063718filed Mar. 19, 2010, and Japanese Patent Application No. 2011-020076filed Feb. 1, 2011.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a photographic subject determinationmethod, to a program product for photographic subject determination, andto a camera.

2. Description of Related Art

An image-capturing device of the following type is per se known. Thisimage-capturing device specifies the position of a photographic subjecton the basis of an AF area selected by the user, and performs processingfor focus adjustment upon this specified subject (refer to JapaneseLaid-Open Patent Publication 2004-205885).

SUMMARY OF THE INVENTION

However, with a prior art image-capturing device, it has not beenpossible to specify the position or the size or the shape of thephotographic subject on the basis of the AF area selected by the user.

According to the 1st aspect of the present invention, a photographicsubject determination method comprises: a binarization step of creatinga plurality of binarized images of a subject image, based upon colorinformation or luminance information in the subject image; an evaluationvalue calculation step of, for each of the plurality of binarizedimages, calculating an evaluation value that is used for specifying atleast one of a position, a size, and a shape of a photographic subjectwithin the subject image; and a photographic subject specification stepof specifying at least one of the position, the size, and the shape of aphotographic subject within the subject image, based upon the evaluationvalue.

According to the 2nd aspect of the present invention, a photographicsubject determination method comprises: a binarization step of creatinga plurality of binarized images of a subject image, based upon colordifference information, luminance information, and color differencespace information combined with the color difference information, in thesubject image; an evaluation value calculation step of, for each of theplurality of binarized images, calculating an evaluation value that isused for specifying at least one of a position, a size, and a shape of aphotographic subject within the subject image; and a photographicsubject specification step of specifying at least one of the position,the size, and the shape of a photographic subject within the subjectimage, based upon the evaluation value.

According to the 3rd aspect of the present invention, in thephotographic subject determination method according to the 1st aspect,it is preferred that the evaluation value includes a first evaluationvalue that is calculated based upon an area of a white pixel region thatis made up by white pixels within a binarized image, and a value thatshows a state of a set of white pixels within the white pixel region.

According to the 4th aspect of the present invention, in thephotographic subject determination method according to the 3rd aspect,it is preferred that the evaluation value includes at least one of asecond evaluation value that is calculated based upon an area of anenveloping rectangle that envelopes the set of the white pixels withinthe binarized image and an area of the set of the white pixels, a thirdevaluation value that is calculated based upon an aspect ratio of theenveloping rectangle, and a fourth evaluation value that is calculatedbased upon a size of a region that includes a face of a person.

According to the 5th aspect of the present invention, in thephotographic subject determination method according to the 4th aspect,it is preferred that in the photographic subject specification step,from among the plurality of white pixel regions, some of the white pixelregions are eliminated based upon the second evaluation value, the thirdevaluation value, and the fourth evaluation value, and, from amongremaining ones of the white pixel regions, the white pixel regions whosefirst evaluation value are large are specified as photographic subjectcandidates.

According to the 6th aspect of the present invention, in thephotographic subject determination method according to the 3rd aspect,it is preferred that the first evaluation value is calculated based uponat least one of the area of the white pixel regions and an inertialmoment centered around a photographic subject inferred position withinthe subject image, an entire area of a screen, and an area of whitepixels that do not correspond to the white pixel regions.

According to the 7th aspect of the present invention, in thephotographic subject determination method according to the 3rd aspect,it is preferred that the value that shows the state of the set of whitepixels within the white pixel region is the area of the white pixelregion and an inertial moment around a photographic subject inferredposition within the image, and the photographic subject inferredposition is either a position designated by a user, or a position inwhich a face of a photographic subject has been detected.

According to the 8th aspect of the present invention, in thephotographic subject determination method according to the 7th aspect,it is preferred that, the photographic subject determination methodfurther comprises an inferring step of inferring a position of an upperhalf of a body of the photographic subject and a position of a lowerhalf of the body of the photographic subject, based upon the position inwhich the face of the photographic subject has been detected.

According to the 9th aspect of the present invention, in thephotographic subject determination method according to the 8th aspect,it is preferred that in the inferring step, a plurality of positions ofthe upper half of the body of the photographic subject and a pluralityof positions of the lower half of the body of the photographic subjectare inferred.

According to the 10th aspect of the present invention, in thephotographic subject determination method according to the 8th aspect,it is preferred that in the photographic subject specification step, atleast one of the position, the size, and the shape of the photographicsubject is specified by combining the white pixel region thatcorresponds to the position of the upper half of the body of thephotographic subject and the white pixel region that corresponds to theposition of the lower half of the body of the photographic subject.

According to the 11th aspect of the present invention, in thephotographic subject determination method according to the 5th aspect,it is preferred that in the photographic subject specification step, atleast one of the position, the size, and the shape of the photographicsubject is specified by combining each of white pixel regions that areproduced by combining a plurality of candidates, among the photographicsubject candidates, whose first evaluation values are large.

According to the 12th aspect of the present invention, in thephotographic subject determination method according to the 1st aspect,it is preferred that in the photographic subject specification step, atleast one of the position, the size, and the shape of the photographicsubject is specified by combining a first white pixel region thatcorresponds to an inferred photographic subject position, and a secondwhite pixel region whose ranging target point is closest to the firstwhite pixel region and that is close thereto upon a screen.

According to the 13th aspect of the present invention, in thephotographic subject determination method according to the 1st aspect,it is preferred that: hue is classified into a plurality ofsubdivisions; and for each subdivision of the plurality of subdivisionsinto which hue has been classified, a binarized image is created bybinarizing the subject image according to pixels that correspond to acorresponding subdivision, and pixels that do not correspond to thecorresponding subdivision, so that a plurality of binarized images thatare created correspond to the plurality of subdivisions of hue.

According to the 14th aspect of the present invention, in thephotographic subject determination method according to the 6th aspect,it is preferred that: a luminance image and a color difference image arecreated based upon luminance information and color information in thesubject image; and binarized images are created for the luminance imageand the color difference image that have been created to be included inthe plurality of binarized images.

According to the 15th aspect of the present invention, a photographicsubject determination method according to the 2nd aspect, it ispreferred that: a luminance image, a first color difference image, and asecond color difference image are created based upon luminanceinformation and color information in the subject image; binarized imagesare created for the luminance image, the first color difference image,and the second color difference image that have been created to beincluded in the plurality of binarized images; and a color differencespace achieved by a first color difference of the first color differenceimage and a second color difference of the second color difference imageis subdivided into a plurality of color difference space subdivisions,and the plurality of binarized images are created by creating binarizedimages corresponding to the color difference space subdivisions, usingthe first color difference image and the second color difference image.

According to the 16th aspect of the present invention, acomputer-readable computer program product comprises a program thatcauses a computer to execute a photographic subject determination methodaccording to the 1st aspect.

According to the 17th aspect of the present invention, acomputer-readable computer program product comprises a program thatcauses a computer to execute a photographic subject determination methodaccording to the 2nd aspect.

According to the 18th aspect of the present invention, a cameracomprises a control unit that performs a photographic subjectdetermination method according to the 1st aspect.

According to the 19th aspect of the present invention, a cameracomprises a control unit that performs a photographic subjectdetermination method according to the 2nd aspect.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the structure of a camera according toan embodiment of the present invention, along with a PC to which thecamera is connected;

FIG. 2 is a figure schematically showing a position in which an areasensed by an AF sensing sensor is located upon a photographic scene;

FIG. 3 is a figure showing a hue circle;

FIG. 4 is a flow chart showing a flow of processing for photographicsubject extraction, in a first embodiment;

FIG. 5 is a figure showing a first concrete example of an image of aphotographic subject;

FIG. 6 is a figure showing a second concrete example of an image of aphotographic subject;

FIG. 7 is a figure showing a concrete example in which the hue circlehas been subdivided into eight sectors on the basis of hue, and eightbinarized images have been created from the subject image, one for eachsector;

FIG. 8 is a figure showing a concrete example when two binarized imageshave been created from each of a Y plane image, a Cb plane image, a Crplane image, and a Y-complement image;

FIG. 9 is a first figure showing a case in which noise elimination hasbeen performed upon a set of binarized images;

FIG. 10 is a second figure showing a case in which noise elimination hasbeen performed upon a set of binarized images;

FIG. 11 is a figure showing a concrete example of a pixel island that iseliminated due to a secondary evaluation value #1;

FIG. 12 is a figure showing a concrete example when pixel islands touchthe left edge and the right edge of the screen are eliminated asphotographic subject candidates;

FIG. 13 is a figure showing a concrete example of photographic subjectcandidates extracted on the basis of a main extraction value;

FIG. 14 is a first figure showing a concrete example of the position,size, and shape of a photographic subject within a subject image;

FIG. 15 is a second figure showing a concrete example of the position,size, and shape of a photographic subject within a subject image;

FIG. 16 is a figure schematically showing processing in the case ofcontinuous shooting photography;

FIG. 17 is a flow chart showing a flow of processing for photographicsubject extraction in a second embodiment;

FIG. 18 is a figure schematically showing a method for binarizing a Yplane image, a Cr plane image, a Cb plane image, and a Y-complementplane image;

FIG. 19 is a figure showing a concrete example of an image of aphotographic subject, in the second embodiment;

FIG. 20 is a figure showing an example of binarization of a Y planeimage, a Cr plane image, a Cb plane image, and a Y-complement planeimage;

FIG. 21 is a figure schematically showing an example in which a colordifference space is subdivided into eight sectors;

FIG. 22 is a figure showing an example of binarization in this colordifference space;

FIG. 23 is a flow chart showing a flow of processing for maskcombination, in this second embodiment;

FIGS. 24A and 24B are figures showing concrete examples of states ofinclusion between two masks;

FIGS. 25A and 25B are figures showing an example of extraction of afirst ranked mask and a second ranked mask in the second embodiment;

FIG. 26 is a figure showing a concrete example of a combined mask thathas been selected by combining the first ranked mask and the secondranked mask, in this second embodiment;

FIG. 27 is a figure showing a concrete example of six binarized imagesthat are used in processing according to a third embodiment;

FIGS. 28A and 28B are figures showing an example of extraction of afirst ranked mask and a second ranked mask in this third embodiment;

FIG. 29 is a figure showing a concrete example of a combined mask thathas been selected by combining the first ranked mask and the secondranked mask, in this third embodiment;

FIG. 30 is a figure showing a concrete example of an image of aphotographic subject, in a fourth embodiment;

FIG. 31 is a figure showing a concrete example of six binarized imagesthat are used in processing according to this fourth embodiment;

FIG. 32 is a figure showing a concrete example of six binarized imagesthat are used in processing according to a fifth embodiment; and

FIG. 33 is a figure showing an example of subject extraction accordingto this fifth embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiment One

FIG. 1 is a block diagram showing the structure of a camera according toan embodiment of the present invention, along with a PC to which thecamera is connected. This camera 100 includes operating members 101, alens 102, an imaging sensor 103, a control unit 104, a memory card slot105, and a monitor 106. The operating members 101 include various inputmembers that are actuated by the user, for example a power supplybutton, a release button, a zoom button, a cruciform key, a confirmbutton, a replay button, a delete button, and so on.

Although the lens 102 actually consists of a plurality of opticallenses, only one lens is shown in FIG. 1 as a representative. Theimaging sensor 103 is, for example, a CCD or a CMOS type image sensor orthe like, and captures an image of a photographic subject that is imagedby the lens 102. And the imaging sensor 103 outputs an image signal tothe control unit 104 based upon this captured image.

On the basis of the imaging signal that is inputted from the imagingsensor 103, the control unit 104 creates image data in a predeterminedimage format, for example in the JPEG format (hereinafter this will betermed the “basic image data”). Moreover, on the basis of this imagedata that has been created, the control unit 104 creates image data fordisplay, for example thumbnail image data. And the control unit 104outputs to the memory card slot 105 an image file that includes thebasic image data and the thumbnail image data that have thus beencreated, with further header information being appended to this file. Inthis embodiment, it will be supposed that both the basic image data andthe thumbnail image data are image data expressed in the RGB colorsystem.

The memory card slot 105 is a slot for insertion of a memory card (notshown in the figures) that serves as a storage medium; the image fileoutputted from the control unit 104 is recorded by being written uponthis memory card. Moreover, according to a command from the control unit104, the memory card slot 105 can be employed for reading in an imagefile that is stored upon the memory card.

The monitor 106 is a liquid crystal monitor that is mounted upon therear surface of the camera 100 (in other words, a rear surface monitor),and images stored upon the memory card and setting menus for setting thecamera 100 and so on are displayed upon this monitor 106. Moreover, whenthe operating mode of the camera 100 is set to photographic mode by theuser, the control unit 104 outputs image data for displaying imagesacquired from the imaging sensor to the monitor 106 in time series. Dueto this, a so-called through image (live view image) is displayed uponthe monitor 106.

The control unit 104 includes a CPU and other peripheral circuitrythereof, and performs overall control of the camera 100. It should beunderstood that SDRAM and flash memory are included in the memoryincluded in the control unit 104. SDRAM is volatile memory and is usedas work memory for holding a program during program execution by theCPU, and is also used as buffer memory for temporarily storing data.Moreover, flash memory is non-volatile memory, and is used for recordingthe program executed by the control unit 104, data for this program, andvarious parameters and so on that are read in during program execution.

The program that is executed by the control unit 104 is stored in theflash memory, as described above, during production in the factory.However, as shown in FIG. 1, it would also be possible to connect thecamera 100 to a personal computer 200 via a USB cable 201, and todownload the program from the personal computer 200. In this case, thisprogram could be supplied to the personal computer 200 via a recordingmedium such as a CD-ROM 202 or the like that is loaded into the personalcomputer 200, or as a data signal via the internet 203 or the like.Moreover, it would also be acceptable to arrange for the program to bestored on a memory card, and to be downloaded into the camera 100 fromthe memory card via the memory card slot 105.

Since the control unit 104 is built as a CPU or the like, the programthat is supplied in this manner is a computer-readable computer programproduct. In this way, the program may be supplied to the camera 100 as acomputer-readable computer program product in any of various formats,such as upon a non-volatile recording medium or as a data signal(including a signal carried upon a carrier wave) or the like.

In this embodiment, the control unit 104 not only specifies the positionof the photographic subject within the image on the basis of theposition of the AF area within the image and on the basis of colorinformation and/or luminance information, but also specifies theposition, the size, and the shape of that photographic subject. Itshould be understood that by the position of the AF area, is meant theposition of the AF area that was selected during photography for use forfocus detection. For example in this embodiment, as shown in FIG. 2, 51AF areas are set upon the photographic scene as a two dimensional array,and corresponds to the various arranged positions of AF sensing sensors.And the control unit 104 displays upon the monitor 106 a plurality ofproposed AF areas that are close to the position of the photographicsubject within the image. When the user selects, from among theseproposed AF areas, that AF area that he has decided is closest to theposition of the photographic subject, then focusing is performed uponthis selected AF area using focus adjustment processing (AF processing)of a per se known type.

Moreover, in this embodiment, among the color information in the image,the hue is used as information for specifying the position, the size,and the shape of the photographic subject. Due to this, the control unit104 first converts the RGB values for each pixel within the subjectimage for which the photographic subject position is to be specified, tohue angle, using the following Equation (1):

$\begin{matrix}{{{Hue}\mspace{14mu} {Conversion}}{H = \left\{ {{{\begin{matrix}{{{60 \times \frac{G - B}{{MAX} - {MIN}}} + 0},} & {{{if}\mspace{14mu} {MAX}} = R} \\{{{60 \times \frac{B - R}{{MAX} - {MIN}}} + 120},} & {{{if}\mspace{14mu} {MAX}} = G} \\{{{60 \times \frac{R - G}{{MAX} - {MIN}}} + 240},} & {{{if}\mspace{14mu} {MAX}} = B}\end{matrix}{if}\mspace{14mu} H} < {0\mspace{14mu} {then}\mspace{14mu} H}} = {H + 180}} \right.}} & (1)\end{matrix}$

It should be understood that the hue is expressed according to the huecircle (color wheel) shown in FIG. 3, and that the hue angle of eachpixel is the angle of its hue upon this hue circle.

In the following, the processing in this embodiment for extraction ofthe photographic subject that has been the subject of focus adjustmentwill be explained using the flow chart shown in FIG. 4. It should beunderstood that the processing shown in FIG. 4 is executed by thecontrol unit 104 as a program that starts when input of image data fromthe imaging sensor 103 is started. In this embodiment, as will bedescribed hereinafter, the details of the processing for an image inwhich detection of a face of a photographic subject is possible, and thedetails of the processing for an image in which detection of a face of aphotographic subject is not possible, are different. Thus, in thefollowing, explanation of the processing for an image in which it ispossible to detect a face of a photographic subject will be explainedusing the image shown in FIG. 5 as an example, while explanation of theprocessing for an image in which it is not possible to detect any faceof a photographic subject will be explained using the image shown inFIG. 6 as an example.

In a first step S100, the control unit 104 reads in the image datainputted from the imaging sensor 103, and then the flow of controlproceeds to a step S200. In this step S200, the control unit 104 reducesthe size of the image data that has been read in so as to enhance thesubsequent processing speed. It should be understood that, if thecontrol unit 104 has sufficient processing capability, then it would beacceptable for this step S200 not to be performed. Next the flow ofcontrol proceeds to a step S300, in which, as described above, thecontrol unit 104 converts the RGB values in the subject image to hueangles, using Equations (1) as described above. Next the flow of controlproceeds to a step S400.

In this step S400, the control unit 104 subdivides the hue circle shownin FIG. 3 into eight sectors (subdivisions), each of angular width 45°.Due to this, the hue circle is divided into a first sector where0°≦hue<45°, a second sector where 45°≦hue<90°, a third sector where90°≦hue<135°, a fourth sector where 135°≦hue<180°, a fifth sector where180°≦hue<225°, a sixth sector where 225°≦hue<270°, a seventh sectorwhere 270°≦hue<315°, and an eighth sector where 315°≦hue<360°.

And, in each of the sectors, the control unit 104 binarizes each pixelwithin the image on the basis of the hue angle of corresponding sector.In other words, for each of the eight sectors, the control unit 104generates a mask image by converting those pixels in the image that havehue within the angular range of that sector into white pixels, whileconverting the other pixels into black pixels. Due to this, for examplefor the image of the photographic subject shown in FIG. 5, eight maskimages are created as shown in FIG. 7: a mask image 7 a for the firstsector, a mask image 7 b for the second sector, . . . , a mask image 7 hfor the eighth sector.

Then the flow of control proceeds to a step S450, in which the controlunit 104 converts the subject image to an image in the YCbCr format, andgenerates each of a image for the Y component (a Y plane image), animage for the Cr component (a Cr plane image), and an image for the Cbcomponent (a Cb plane image). Moreover, the control unit 104 inverts thewhite and black pixels in the Y plane image to generate a Y-complementplane image. In concrete terms, using the following Equations (2)through (4), the control unit 104 converts the subject image that isexpressed in the RGB color system into a luminance image that consistsof the luminance component (i.e. the Y component) and color difference(chrominance) images that consist of the color difference (chrominance)components (the Cb component and the Cr component) in the YCbCr colorspace.

In other words, for the subject image, the control unit 104 creates aluminance image as a Y plane image that consists of the Y componentusing the following Equation (2), and creates a color difference imagethat consists of the Cb component and a color difference image thatconsists of the Cr component as a Cb plane image and a Cr plane imagerespectively, using the following Equations (3) and (4):

Y=0.299R+0.587G+0.114B  (2)

Cb=−0.169R−0.332G+0.500B  (3)

Cr=0.500 R−0.419G−0.081B  (4)

And for each of the Y plane image, the Cb plane image, the Cr planeimage, and the Y-complement plane image that have thus been created, thecontrol unit 104 compares the density values of each pixel in the imagesand calculates the average of the density values and the standarddeviation of the density; and then the flow of control proceeds to astep S500.

In this step S500, the control unit 104 creates four first binarizedimages by binarizing the pixels of the Y plane image, the Cb planeimage, the Cr plane image, and the Y-complement plane image respectivelyby the corresponding average values, and four second binarized images bybinarizing the pixels of each of these images are binarized by thecorresponding average values plus one standard deviation. Due to this,for example for the image of the photographic subject shown in FIG. 5,as shown in FIG. 8, a first binarized image 8 a and a second binarizedimage 8 b are created for the Y plane image, a first binarized image 8 cand a second binarized image 8 d are created for the Cb plane image, afirst binarized image 8 c and a second binarized image 8 f are createdfor the Cr plane image, and a first binarized image 8 g and a secondbinarized image 8 h are created for the Y-complement plane image.

Then the flow of control proceeds to a step S600, in which the controlunit 104 performs face detection processing using an inbuilt facedetection function, and makes a decision as to whether or not any facehas been detected within the subject image. For example, the controlunit 104 may perform per se known face recognition processing upon thesubject image, and the result will be a decision as to whether or notany face of a person has been detected within the subject image. If anegative decision is reached in this step S600, in other words if, asshown in the FIG. 6 example, the subject image is an image in which itis not possible to detect any human face, then the flow of controlproceeds to a step S700. In this step S700, the center position withinthe subject image, or the focus adjustment position (i.e. the AFposition) within the subject image, is set as the inferred position ofthe photographic subject (i.e. as the inferred position point of thephotographic subject), and then the flow of control is transferred to astep S1000 that will be described hereinafter.

By contrast, if an affirmative decision is reached in the step S600, inother words if, as shown in the FIG. 5 example, the subject image is animage in which it is possible to detect a human face, then the flow ofcontrol is transferred to a step S800. In this step S800, as shown inFIG. 5, the control unit 104 specifies the region 5 a that includes theface that has been detected by the above face detection processing, andestimates the position of the upper half of the body of the photographicsubject and the position of the lower half of his or her body on thebasis of the size of the region 5 a that has thus been specified.

For example the control unit 104 may estimate, as being the position ofthe upper half of the body of the photographic subject, the followingthree points: a point 5 b that is shifted from the center of thespecified region 5 a in the downwards direction by a predetermineddistance, and points 5 c and 5 d at a predetermined distance in thedownwards direction from the vertical framing lines that surround theregion 5 a. Moreover the control unit 104 may estimate, as being theposition of the lower half of the body of the photographic subject, thefollowing four points: points 5 e and 5 f that are shifted from theabove described point 5 b in the downwards direction by predetermineddistances, a point 5 g that is shifted from the above described point 5c in the downwards direction by a predetermined distance, and a point 5h that is shifted from the above described point 5 d in the downwardsdirection by a predetermined distance. In this manner, the estimatedpoints 5 b through 5 d for the position of the upper half of the body ofthe photographic subject within the image and the estimated points 5 ethrough 5 h for the position of the lower half of his or her body withinthe image are set. It should be understood that, in this embodiment, theestimated points 5 b through 5 d for the position of the upper half ofthe body of the photographic subject will be termed the “inferredposition points of the photographic subject”.

Then the flow of control proceeds to a step S900, in which the controlunit 104 calculates evaluation values that are used by the subsequentprocessing on the basis of the area of the region 5 a that has beenspecified by the face recognition processing above. For example, thecontrol unit 104 may calculate two evaluation values by multiplying thearea of the region 5 a by predetermined multiples, for example by 0.5and 2.0. It should be understood that the evaluation values that arecalculated here will be termed the “secondary evaluation values #3”, inorder to distinguish them from other evaluation values that arecalculated by the subsequent processing. Then the flow of controlproceeds to a step S1000.

In the step S1000, the control unit 104 selects one from the sixteenbinarized images shown in FIGS. 7 and 8, and performs noise eliminationupon this selected binarized image using a median filter. For example,the binary images shown in FIG. 9 may be obtained as the result ofperforming noise elimination upon the images shown in FIG. 7, and thebinary images shown in FIG. 10 may be obtained as the result ofperforming noise elimination upon the images shown in FIG. 8. Subsequentprocessing is executed upon this binarized image for which noiseelimination has been performed.

The flow of control then proceeds to a step S1100, in which the controlunit 104 performs labeling processing upon the binarized image selectedin the step S1000 and for which noise elimination has been performed. Inconcrete terms, the control unit 104 performs this labeling processingas follows. First, the control unit 104 extracts unified sets of whitepixels and unified sets of black pixels from within the binarized imageas being labeling regions, and, among these extracted labeling regions,detects those labeling regions that consist of white pixels as beingpixel islands.

Then the flow of control proceeds to a step S1200, in which the controlunit 104 calculates the area of each of the pixel islands that have beendetected within the binary image, and then the flow of control proceedsto a step S1300. In this step S1300, the control unit 104 takes a pixelisland that has been detected within the binarized image as a subject,and calculates the inertial moment of this pixel island by taking theinferred position point of the photographic subject that was set in thestep S700 or in the step S800 as a center, (i.e., the inertial moment ofits white pixels around the barycenter). It should be understood that,while detailed explanation of the method by which the inertial moment inthe binarized image is calculated is herein omitted since it is per seconventional, for example, it would be possible to calculate thisinertial moment by summing the squares of the pixel distances from theinferred position point of the photographic subject multiplied by (0 or1). Then the flow of control proceeds to a step S1400.

In the step S1400 the control unit 104 eliminates, from within thebinarized image, pixel islands that are larger than some fixed size, forexample pixel islands the ratio of whose area to that of the entirebinarized image is 60% or greater, and pixel islands that are smallerthan some fixed size, for example pixel islands the ratio of whose areato that of the entire binarized image is 1% or less. After this the flowof control proceeds to a step S1500, in which the control unit 104 takesas subjects the pixel islands that remain as the result of eliminationin the step S1400, and, calculates a “main evaluation value” for eachpixel island used for specifying the position of the photographicsubject within the subject image and also for specifying the position,the size, and the shape of the photographic subject in the subjectimage, on the basis of the inertial moment of the white pixels aroundthe barycenter calculated in the step S1300, according to the followingEquation (5):

main evaluation value=number of white pixels constituting the pixelisland/moment of inertia of the white pixels around the barycenter ascenter  (5)

Then the flow of control proceeds to a step S1600, in which the controlunit 104 sets, for each of the pixel islands, an enveloping rectanglethat encloses that pixel island, and then calculates a “secondaryevaluation value #1” for each of the pixel islands according to thefollowing Equation (6):

secondary evaluation value #1=white pixel area/area of envelopingrectangle  (6)

These secondary evaluation values #1 are values for eliminating pixelislands that have wavy in and out contours or that have a lot of voidportions and that therefore are not ones normally found in aphotographic subject, such as the pixel island shown by way of examplein FIG. 11; and, in a step S1720 that will be described subsequently, ifthese values are less than or equal to a predetermined threshold value(for example 0.2), then the control unit 104 eliminates these pixelislands from the subjects for subsequent processing.

Then the flow of control proceeds to a step S1700 in which, asevaluation values that are used for eliminating, from among the pixelislands, those ones that are long and narrow and therefore cannot benormal photographic subjects, the control unit 104 calculates for eachof the pixel islands, as a “secondary evaluation value #2”, the aspectratio of the rectangular envelope that was set in the step S1600. In thenext step S1720 that will be described hereinafter, those pixel islandsfor which this value is within a predetermined range, for example therange in which it is greater than or equal to 0.2 and less than 5, aretaken as ones that are long and narrow so that they cannot be normalphotographic subjects, and thus are eliminated as subjects for thesubsequent processing. Then the flow of control proceeds to the stepS1720.

In this step S1720, the control unit 104 eliminates certain ones of thepixel islands included in the binarized image, using the above described“secondary evaluation values #1” and “secondary evaluation values #2”.In other words, as described above, among the various pixel islands, thecontrol unit 104 eliminates subjects for the subsequent processing byeliminating, as photographic subject candidates, those pixel islands forwhich the “secondary evaluation value #1” is less than or equal to thepredetermined threshold value, for example 0.2, and those pixel islandsfor which the “secondary evaluation value #2” is within thepredetermined range, for example that are greater than or equal to 0.2or less than 5. Then the flow of control proceeds to a step S1750.

In the step S1750, the control unit 104 eliminates from the photographicsubject candidates those pixel islands that touch the left edge or theright edge of the screen. For example if, as shown in FIG. 12, thenumber of pixels in a pixel island at either the left edge or the rightedge is greater than or equal to ⅓ of the number of pixels vertically,then this pixel island is eliminated. Due to this, in the binarizedimage shown in FIG. 12, the pixel island 12 a that touches the left edgeof the image and the pixel island 12 b that touches its right edge areboth eliminated. Then the flow of control proceeds to a step S1800.

In the step S1800, the control unit 104 makes a decision as to whetheror not it was possible to detect a face in the subject image, as theresult of the decision in the step S600 described above. If a negativedecision is reached in this step S1800, then the flow of control istransferred to a step S2000. By contrast, if an affirmative decision isreached in this step S1800, then the flow of control proceeds to a stepS1900. In this step S1900, the control unit 104 eliminates certain onesof pixel islands included in the binarized image, using the secondaryevaluation values #3 that were calculated in the step S900. For example,using the two secondary evaluation values #3 calculated by multiplyingthe area of the region 5 a by predetermined magnifications, for exampleby 0.5 and by 2 as described above, the control unit 104 may eliminate,as photographic subject candidates, those pixel islands whose areas areless than the secondary evaluation value #3 calculated by multiplyingthe area of the region 5 a by 0.5, and those pixel islands whose areasare greater than or equal to the secondary evaluation value #3calculated by multiplying the area of the region 5 a by 2. By doingthis, it is possible to eliminate pixel islands that are too big, andpixel islands that are too small, to be capable of being thephotographic subject.

Then the flow of control proceeds to the step S2000 in which, taking assubjects the remaining pixel islands from among the pixel islandsincluded in the binarized image after elimination using the secondaryevaluation values #1 through #3, the first ranked pixel island for whichthe main evaluation value is the largest and the second ranked islandfor which the main evaluation value is the second largest are taken asphotographic subject candidates for this binarized image and areextracted, and then the flow of control proceeds to a step S2100. Inthis step S2100, the control unit 104 decides whether or not theprocessing from the step S1000 through the step S2000 has been completedfor all of the 16 binarized images shown in FIGS. 7 and 8. If a negativedecision is reached in this step S2100, then the control unit 104selects one of the binarized images that has not yet been processed, andrepeats the above processing. But if an affirmative decision is reachedin this step S2100, then the flow of control proceeds to a step S2200.

In this step S2200, the control unit 104 decides, on the basis of theresult of the decision in the step S600 described above, whether or notit was possible to detect a face in the subject image. If an affirmativedecision is reached in this step S2200, then the flow of control istransferred to a step S2500, in which, for each of the binarized images,the control unit 104 selects from among the photographic subjectcandidates that were extracted in the step S2000, in other words fromthe pixel island the magnitude of whose main evaluation value is rankedfirst and the pixel island the magnitude of whose main evaluation valueis ranked second, those pixel islands that respectively correspond tothe positions of the three points (the points 5 b through 5 d) for theupper half of the body and the four points (5 e through 5 h) for thelower half of the body. Due to this, as for example shown in FIG. 13,the pixel islands 13 a through 13 c in the binarized image 10 b thatcorrespond to the lower half of the body of the photographic subject,the pixel islands 13 d through 13 h in the binarized image 10 c thatindicate the upper half of his or her body, and the pixel islands 13 ithrough 13 m in the binarized image 9 d that indicate the upper half ofhis or her body, are selected.

Then the flow of control proceeds to a step S2600, in which the controlunit 104 combines the pixel islands of the binarized images 10 b, 10 c,and 9 d that were selected in the step S2500, and thereby the shape ofthe photographic subject within the subject image is extracted. Thiscombination may take, for example, the logical sum (OR) of their whitepixels. By doing this, as shown in FIG. 14, the position and the shapeof the photographic subject 14 a within the subject image are extracted.Moreover, by doing this, the size of the photographic subject 14 awithin the subject image is also specified. Then the flow of controlproceeds to a step S2700, in which the control unit 104 records in thememory the positions of the barycenters of the pixel islands within thesubject image that were selected in the step S2500, and the number ofbinarized images that were employed in this combination, and then thisprocessing terminates.

On the other hand, if the result of the decision in the step S2200 isnegative, then the flow of control is transferred to a step S2300, inwhich the control unit 104 takes that first ranked pixel island (i.e.that first pixel island) extracted in the step S2000 whose mainevaluation value is the greatest as being the photographic subjectestimated point, and calculates the ranging value of the neighborhood ofthat photographic subject estimated point. Then the flow of controlproceeds to a step S2400, in which, for each of the binarized images,the control unit 104 specifies a first pixel island, and also specifies,as being a second pixel island, the pixel island for which the rangingtarget point is closest to the ranging target point of that first pixelisland and that moreover is close upon the screen thereto. And thecontrol unit 104 specifies the position, the size, and the shape of thephotographic subject within the subject image by combining the binarizedimage in which the first pixel island is extracted and the binarizedimage in which the second pixel island is extracted. For example if, asshown in FIG. 15, a first pixel island 15 b is extracted from within thebinarized image 15 a and a second pixel island 15 d is extracted fromwithin the binarized image 15 c, then the position, the size, and theshape of the photographic subject 15 f can be specified on the basis ofthe combined image 15 e that is obtained by combining the binarizedimage 15 a and the binarized image 15 c.

According to the first embodiment of the present invention as explainedabove, the following beneficial operational advantages are obtained.

(1) The control unit 104 classifies the subject image into onescorresponding to eight sectors (subdivisions) on the basis of hue angle,and binarizes an image in each of the sectors. Furthermore, the controlunit 104 also binarizes the luminance image and the two color differenceimages, and further obtains a binarized Y-complement image by invertingthe binarized luminance image. Moreover it is arranged for the controlunit 104 to calculate evaluation values that are employed for specifyingthe position, the size, and the shape of the photographic subject withinthe subject image on the basis of these binarized images, and to specifythe position, the size, and the shape of the photographic subject withinthe subject image on the basis of these evaluation values. Due to this,it is possible to specify the position, the size, and the shape of thephotographic subject within the subject image at high accuracy.

(2) It is arranged for the evaluation values to include the mainevaluation values that are calculated according to Equation (5). Due tothis, it is possible to specify the position, the size, and the shape ofthe photographic subject at high accuracy while taking into account theareas of the pixel islands, and the overall states of the white pixelswithin the pixel islands.

(3) It is arranged for the evaluation values to include the secondaryevaluation values #1 that are calculated according to Equation (6), thesecondary evaluation values #2 that are calculated for each of the pixelislands on the basis of the aspect ratio of the enveloping rectanglethat encloses that pixel island, and the secondary evaluation values #3that are calculated on the basis of the sizes of the regions thatinclude the face of a person. Due to this, it is possible to eliminatein advance, from the subjects for processing, those pixel islands whoseshapes or sizes cannot be those of normal photographic subjects.

(4) From among the plurality of pixel islands, it is arranged for thecontrol unit 104 to eliminate from the subjects of processing thosepixels islands for which, on the basis of the secondary evaluationvalues #1 through #3, the possibility of being the photographic subjectis low, and, from the remainder of the pixel islands, to specify as thephotographic subject candidates those whose main evaluation values arelarge. By doing this, it is possible to specify the photographic subjectcandidates with high accuracy.

(5) It is arranged for the control unit 104 to set, as the inferredposition of the photographic subject, either a position that has beendesignated by the user, or a position at which the face of aphotographic subject has been detected. Due to this, it is possible toinfer the position of the photographic subject by simple processing.

(6) It is arranged for the control unit 104 to infer the position of theupper half of the body of the photographic subject and the position ofthe lower half of his body on the basis of the position at which theface of the photographic subject has been detected. Due to this, it ispossible to infer the position of the upper half of the body of thephotographic subject and the position of the lower half of his body withsimple processing that takes the position of the face of thephotographic subject as a reference.

(7) It is arranged to specify the position, the size, and the shape ofthe photographic subject by combining pixel islands that correspond tothe position of the upper half of the body of the photographic subjectand pixel islands that correspond to the position of the lower half ofhis or her body. By doing this, if it is possible to detect the face ofthe photographic subject, then it is possible to specify the position,the size, and the shape of the photographic subject at high accuracy.

(8) It is arranged for the control unit 104 to specify the first pixelisland whose main evaluation value is the largest, and also, as thesecond pixel island, the pixel island whose ranging target point isclosest to that of the first pixel island and that moreover is in aposition close thereto upon the screen, and to specify the position, thesize, and the shape of the photographic subject by combining these. Bydoing this, it is still possible to specify the position, the size, andthe shape of the photographic subject at high accuracy, even if it isnot possible to detect the face of the photographic subject.

Embodiment Two

In the first embodiment described above, an example was explained inwhich the hue in the color information for the image was used as theinformation for specifying the position, the size, and the shape of thephotographic subject. By contrast, in this second embodiment, an exampleis explained in which the control unit 104 uses the luminance, the colordifference (chrominance) and the color difference (chrominance) space inthe color information for the image, as the information for specifyingthe position, the size, and the shape of the photographic subject.

FIG. 17 is a figure showing a flow of processing in a second embodimentfor extraction of a photographic subject that is a subject for focusadjustment. The processing shown in FIG. 17 is executed by the controlunit 104 as a program that starts when input of image data from theimaging sensor 103 is started. In a first step S3000, the control unit104 reads in the image data inputted from the imaging sensor 103, andthen the flow of control proceeds to a step S3100. In this step S3100,input is received from the user of the general position of thephotographic subject within the image data that has been read in thestep S3000. It is possible to extract the photographic subjectaccurately, if the input of the position of the photographic subject isreceived from the user. However, if the accuracy by which extraction ofthe photographic subject can be performed is not considered, the controlunit 104 could set some specified position within the image, such as forexample the position of the center of the image, as the position of thephotographic subject without receiving the designation of thephotographic subject from the user.

Then the flow of control proceeds to a step S3200, in which, in asimilar manner to the first embodiment, the control unit 104 convertsthe subject image to an image in the YCbCr format, an creates each of aY plane image, a Cr plane image, a Cb plane image, and a Y-complementplane image. And, for each of the Y plane image, the Cr plane image, theCb plane image, and the Y-complement plane image, the control unit 104calculates the average value Ave of its pixel values and their standarddeviation a. Then the flow of control proceeds to a step S3300, in whichthe control unit 104 creates a two dimensional color difference space (aCb—Cr space) having the Cb value along the vertical axis and the Crvalue along the horizontal axis, and then the flow of control proceedsto a step S3400.

In this step S3400, the control unit 104 binarizes each of the Y planeimage, the Cr plane image, the Cb plane image, and the Y-complementplane image using as threshold values the average value Ave of itspixels, and the standard deviation σ, as shown in FIG. 18. Due to this,on the basis of the subject image shown in FIG. 19, four binarizedimages are created for each of the Y plane image, the Cr plane image,the Cb plane image, and the Y-complement plane image; i.e., in total, 16binarized images are created, as shown in FIG. 20.

Then the flow of control proceeds to a step S3500 in which, using theimage data that has been converted to the YCbCr format, the control unit104 binarizes the subject image in eight ways, according to eightsectors in the color difference space that was created in the stepS3300. In concrete terms, as shown in FIG. 21, the control unit 104subdivides the two dimensional color difference space (a Cb—Cr space)having the Cb value along the vertical axis and the Cr value along thehorizontal axis at equal angular intervals into the eight sectors(subdivisions) 21 a through 21 h. And then, as an initial step, thecontrol unit 104 creates eight images each having the same size as thesubject image, corresponding to each of these sectors 21 a through 21 h,and sets all of the pixel values in each of these eight binary images to0.

Then, using the pixel values of the corresponding pixels of the Cb planeimage and the Cr plane image, the control unit 104 binarizes the subjectimage into the binary images corresponding eight sectors 21 a through 21h, using the following Equations (7) through (14). In other words, thecontrol unit 104 binarizes the subject image into each of the binaryimages in each of the sectors 21 a through 21 h on the basis of themagnitude relationships of the Cb values and the Cr values, the sign ofthe Cb values and the sign of the Cr values in the subject image, asfollows:

Cb≧0,Cr≧0, and |Cr|≧|Cb|→sector 21a  (7)

Cb≧0,Cr≧0, and |Cr|<|Cb|→sector 21b  (8)

Cb≧0,Cr<0, and |Cr|≦|Cb|→sector 21c  (9)

Cb≧0,Cr<0, and |Cr|>|Cb|→sector 21d  (10)

Cb<0,Cr<0, and |Cr|>|Cb|→sector 21e  (11)

Cb<0,Cr<0, and |Cr|≦|Cb|→sector 21f  (12)

Cb<0,Cr≧0, and |Cr|<|Cb|→sector 21g  (13)

Cb<0,Cr≧0, and |Cr|≧|Cb|→sector 21h  (14)

In concrete terms, if the pixel values of the corresponding pixels inthe Cb plane image and in the Cr plane image satisfy Equation (7), thenthe control unit 104 changes the pixel value of the corresponding pixelin the binary image of sector 21 a to 1. Moreover, if the pixel valuesof the corresponding pixels in the Cb plane image and in the Cr planeimage satisfy Equation (8), then the control unit 104 changes the pixelvalue of the corresponding pixel in the binary image of sector 21 bto 1. In a similar manner, if the pixel values of the correspondingpixels in the Cb plane image and in the Cr plane image satisfy Equation(9), then the control unit 104 changes the pixel value of thecorresponding pixel in the binary image of sector 21 c to 1. And, if thepixel values of the corresponding pixels in the Cb plane image and inthe Cr plane image satisfy Equation (10), then the control unit 104changes the pixel value of the corresponding pixel in the binary imageof sector 21 d to 1.

Furthermore, if the pixel values of the corresponding pixels in the Cbplane image and in the Cr plane image satisfy Equation (11), then thecontrol unit 104 changes the pixel value of the corresponding pixel inthe binary image of sector 21 e to 1; and, if the pixel values of thecorresponding pixels in the Cb plane image and in the Cr plane imagesatisfy Equation (12), then the control unit 104 changes the pixel valueof the corresponding pixel in the binary image of sector 21 f to 1.Moreover, if the pixel values of the corresponding pixels in the Cbplane image and in the Cr plane image satisfy Equation (13), then thecontrol unit 104 changes the pixel value of the corresponding pixel inthe binary image of sector 21 g to 1; and, if the pixel values of thecorresponding pixels in the Cb plane image and in the Cr plane imagesatisfy Equation (14), then the control unit 104 changes the pixel valueof the corresponding pixel in the binary image of sector 21 h to 1.

By performing the decision described above for all of the correspondingpixels in the Cb plane image and in the Cr plane image by usingEquations (7) through (14) described above, the control unit 104 createseight binarized images on the basis of the eight sector images. Due tothis, the binarized images of eight sectors shown in FIG. 22 are createdon the basis of the subject image shown in FIG. 19.

Then the flow of control proceeds to a step S3600. The processing fromthe step S360 to the step S4100 is executed for each of the total of 24binarized images, consisting of the binarized images correspondingsixteen subdivisions shown in FIG. 20 and the binarized imagescorresponding eight subdivisions shown in FIG. 22. In this step S3600,the control unit 104 takes as subject a single binarized image that hasbeen selected from among the total of 24 binarized images consisting ofthe 16 binarized images shown in FIG. 20 and the eight binarized imagesshown in FIG. 22, and performs labeling processing upon that singlebinarized image by recognizing the state in which the pixels in thatbinarized image are linked together. For this labeling processing, a perse known method may be used. For example, as labeling regions, thecontrol unit 104 may extract aggregated sets of white pixels andaggregated sets of black pixels within the binarized image, may detectlabeling regions consisting of white pixels among these labeling regionsthat have been extracted as being pixel islands, and may performlabeling by attaching labels to these pixel islands.

Then the flow of control proceeds to a step S3700, in which the controlunit 104 makes a decision as to whether or not there is any pixel island(i.e. a clump of white pixels) to which a label has been attached. If anegative decision is reached in this step S3700, then the flow ofcontrol is transferred to a step S4200 that will be describedhereinafter. But if an affirmative decision is reached in this stepS3700, then the flow of control proceeds to a step S3800. In this stepS3800, the control unit 104 calculates the white pixel area of eachpixel island that has been labeled. Then the flow of control proceeds toa step S3900, in which the control unit 104 takes as subject the whitepixels of each pixel island that has been labeled, and calculates theirinertial moment (the inertial moment of the white pixels around thebarycenter) around the position of the photographic subject that wasdesignated by the user in the step S3100 as a center. It should beunderstood that detailed explanation of the method of calculating theinertial moment of the white pixels in the binarized image around thebarycenter is omitted because it is per se known; for example, it wouldbe possible to calculate this inertial moment by summing the squares ofthe pixel distances from the inferred position point of the photographicsubject multiplied by (0 or 1). Then the flow of control proceeds to astep S4000.

In this step S4000, the control unit 104 calculates a main evaluationvalue according to the following Equation (15), on the basis of theinertial moment of the white pixels around the barycenter that has beencalculated in the step S3900.

main evaluation value=number of white pixels making up the pixelisland/inertial moment of these white pixels around the barycenter as acenter  (15)

Then the flow of control proceeds to a step S4100, in which the controlunit 104 extracts a first ranked pixel island for which the mainevaluation value calculated in the step S400 is largest, and a secondranked pixel island for which it is second largest, as photographicsubject candidates for this binarized image, and then the flow ofcontrol proceeds to a step S4200. In this step S4200, the control unit104 decides whether or not the processing from the step S3600 throughthe step S4100 has been completed for all of the total of 24 binarizedimages, i.e. the 16 binarized images shown in FIG. 20 and the eightbinarized images shown in FIG. 22. If a negative decision is reached inthis step S4200, then the flow of control returns to the step S3600, oneof the binarized images among those binarized images for which theprocessing has not yet been performed is selected, and the processingfrom the step S3600 through the step S4100 is executed again. But if anaffirmative decision is reached in this step S4200, then the flow ofcontrol proceeds to a step S4300.

In this step S4300, the control unit 104 compares together the mainevaluation values for the first ranked pixel islands and the secondranked pixel islands that have been extracted from each of the 24binarized images, in other words compares together the main evaluationvalues of a total of 48 pixel islands, and extracts the first rankedpixel island and the second ranked pixel island in the entire total of24 binarized images. In order to distinguish this first ranked pixelisland and this second ranked pixel island for the entire set of 24binarized images that are extracted in this step S4300 from the firstranked pixel islands and the second ranked pixel islands that wereextracted from each of the binarized images in the earlier step S4100,they will be termed the “first ranked mask” and the “second rankedmask”. Then the flow of control proceeds to a step S4400, in which thecontrol unit 104 calculates the coordinates of the barycenter of each ofthe first ranked pixel island and the second ranked pixel island amongthe entire set thereof, and the coordinates of enveloping frames foreach of those pixel islands, and then the flow of control proceeds to astep S4500.

In the step S4500, the control unit performs mask combination processingshown in FIG. 23. Now, this mask combination processing will beexplained. In a step S4510, the control unit calculates a “Mask Rate”according to the following Equation (16):

Mask Rate=area of second ranked mask/area of first ranked mask  (16)

Then the flow of control proceeds to a step S4520, in which the controlunit 104 makes a decision as to whether or not Mask Rate is within apredetermined range, for example between 0.33 and 3. If a negativedecision is reached in this step S4520, then the flow of control istransferred to a step S4540 that will be described hereinafter. But ifan affirmative decision is reached in this step S4520, then the flow ofcontrol proceeds to a step S4530. The reason that, in this manner, theprocessing of the step S4530 is only executed if Mask Rate is within thepredetermined range, is in order to prevent combination of a firstranked mask and a second ranked mask whose sizes are too different.

In the step S4530, the control unit 104 makes a decision as to whetheror not there is any overlapping portion between the enveloping framethat envelopes the first ranked mask and the enveloping frame thatenvelopes the second ranked mask. If a negative decision is reached inthis step S4530, then the flow of control proceeds to the step S4540.And, in this step S4540, the control unit 104 selects the first rankedmask as being the combined mask, and then the flow of control returns tothe processing shown in FIG. 17.

By contrast, if an affirmative decision is reached in the step S4520,then the flow of control is transferred to a step S4550. In this stepS4550, the control unit 104 makes a decision as to whether or not one ofthe first ranked mask and the second ranked mask is included in theother. For example if, as shown in FIG. 24A, the result of subtractingthe second ranked mask from the first ranked mask includes no pixels ofvalue −1, then the control unit 104 may decide that the first rankedmask fully includes the second ranked mask. On the other hand, thecontrol unit 104 may decide that the second ranked mask fully includesthe first ranked mask, if the result of subtracting the second rankedmask from the first ranked mask includes no pixels of value +1.

Moreover if, as shown in FIG. 24B, the result of subtracting the secondranked mask from the first ranked mask includes both at least one pixelof value of +1 and at least one pixel of value −1, then the control unit104 calculates the numerical value obtained by dividing the number ofpixels of value +1 by the number of pixels in the first ranked mask(hereinafter termed the “numerical value #1”), and the numerical valueobtained by dividing the number of pixels of value −1 by the number ofpixels in the second ranked mask (hereinafter termed the “numericalvalue #2”). And, if the numerical value #1 is greater than or equal tothe numerical value #2, and moreover the numerical value #2 is smallerthan some predetermined value, for example 0.05, then the control unit104 decides that the first ranked mask partially includes the secondranked mask. On the other hand, if the numerical value #2 is greaterthan or equal to the numerical value #1, and moreover the numericalvalue #1 is smaller than the predetermined value, for example 0.05, thenthe control unit 104 decides that the second ranked mask partiallyincludes the first ranked mask. By doing this, if either one of themasks includes 95% or more of the other, then this is taken as beinginclusion (full inclusion or partial inclusion).

If an affirmative decision is reached in the step S4550, then the flowof control is transferred to a step S4570, in which the control unit 104selects the larger one among the first ranked mask and the second rankedmask as being the combined mask, and then the flow of control returns tothe processing of FIG. 17. But if a negative decision is reached in thestep S4550, then the flow of control proceeds to a step S4560, in whichthe control unit 104 combines the first ranked mask and the secondranked mask by taking their logical sum (OR) and selects the result asthe combined mask, and then the flow of control returns to theprocessing shown in FIG. 17. For example, if the first ranked mask isthe mask 25 a shown in FIG. 25A and the second ranked mask is the mask25 b shown in FIG. 25B, then these two masks are combined by logicaladdition (OR), and the result, i.e. the mask 26 a shown in FIG. 26, isselected as being the combined mask. By doing this, it is possible toextract the combined mask 26 a that gives the shape of the photographicsubject within the subject image shown in FIG. 19, and, on the basis ofthis combined mask 26 a, it is possible to specify the position, thesize, and the shape of the photographic subject within the subjectimage.

In the step S4600 of the FIG. 17 flow chart, as information related tothe mask that has been selected as the combined mask, the control unit104 outputs the barycenter of the mask that has been selected as thecombined mask, the mask number of the combined mask (in other words, thelabel number that was assigned during the labeling processing), and thecoordinate values of the four end points of the enveloping frame thatenvelopes the combined mask, and then the processing of this flow chartterminates.

According to the second embodiment as explained above, it is arrangedfor the control unit 104 to binarize the subject image using theluminance, the color differences, and the color difference space in thecolor information of the image, to calculate evaluation values that areused for specifying the position, the size, and the shape of thephotographic subject within the subject image on the basis of thesebinarized images, and to specify the position, the size, and the shapeof the photographic subject within the subject image on the basis ofthese evaluation values. Due to this, it is possible to specify theposition, the size, and the shape of the photographic subject within thesubject image with yet further accuracy.

Embodiment Three

In the second embodiment of the present invention described above anexample was explained in which a combined mask was extracted byperforming the described processing while taking as subjects a total of24 binarized images, consisting of the sixteen binarized images shown inFIG. 20 and the eight binarized images shown in FIG. 22, and thereby theposition, the size, and the shape of the photographic subject werespecified within the subject image. In this case, since the noise innumbers 4 through 12 and 16 of the binarized images shown in FIG. 20 ishigh, accordingly it is considered that the possibility is low that amask will emerge from analysis of these noisy binarized images that canbe used for extraction of the photographic subject. Due to this, thereis a high probability that after processing has been performed uponthese ones of the binarized images it will turn out to have beenuseless, and this is undesirable.

Thus, in this third embodiment, among the 16 binarized images shown inFIG. 20, processing is performed to take a total of 14 binarized imagesas subjects: the eight binarized images shown in FIG. 22; and the sixbinarized images shown in FIG. 27, in other words the three binarizedimages that were binarized from the Y plane image, the Cr plane image,and the Cb plane image so that their pixels whose pixel values weregreater than or equal to their average value Ave+ their standarddeviation a became white pixels, and the three binarized images thatwere binarized from the Y plane image, the Cr plane image, and the Cbplane image so that their pixels whose pixel values were less than orequal to their average value Ave− their standard deviation a becamewhite pixels.

By doing this, in this third embodiment, the first ranked mask 28 ashown in FIG. 28A and the second ranked mask 18 b shown in FIG. 28B areextracted on the basis of the subject image shown in FIG. 19, and themask 29 a shown in FIG. 29 resulting from these two being logicallyadded (ORed) is selected as the combined mask. Due to this, it ispossible to extract the combined mask 29 a giving the shape of thephotographic subject within the subject image shown in FIG. 19, and itis possible to specify the position, the size, and the shape of thephotographic subject on the basis of this combined mask 29 a. While,with the processing according to this third embodiment, it is possibleto obtain similar results to those obtained with the second embodiment,since it is possible to reduce the number of binarized images that areemployed for the processing to fewer than in the case of the secondembodiment, accordingly it is possible to increase the speed of theprocessing.

Embodiment Four

Furthermore, it would also be acceptable to arrange to perform theprocessing using only the six binarized images shown in FIG. 27. Forexample, it would be possible for the control unit 104 to create the sixbinarized images shown in FIG. 31 on the basis of the subject imageshown in FIG. 30, and to specify the position, the size, and the shapeof the photographic subject within the subject image, in other words theposition, the size, and the shape of the jet coaster, on the basis ofthese binarized images. Since, with this method, it is possible yetfurther to reduce the number of binarized images that are used in theprocessing, accordingly it becomes possible to increase the processingspeed yet further.

Embodiment Five

In the method of the fourth embodiment in which the six binarized imagesdescribed above are employed, sometimes it is difficult to specify theposition, the size, and the shape of the photographic subject accuratelywhen the background and the photographic subject are similar in color.In this type of case it will be effective, as shown in FIG. 32, to addthree more binarized images to the analysis procedure, thus employing atotal of nine binarized images.

In concrete terms, as shown in FIG. 32, the control unit 104 performsits processing by adding, to the above described three binarized imagesthat were binarized from the Y plane image, the Cr plane image, and theCb plane image so that their pixels whose pixel values were greater thanor equal to their average value Ave+ their standard deviation a becamewhite pixels and the above described three binarized images that werebinarized from the Y plane image, the Cr plane image, and the Cb planeimage so that their pixels whose pixel values were less than or equal totheir average value Ave− their standard deviation a became white pixels,an additional three binarized images that are binarized therefrom sothat their pixels whose pixel values are greater than or equal to theiraverage value Ave−1.6× their standard deviation a become white pixels.By doing this, as shown in FIG. 33, due to the pixel island 33 a that isextracted from within the binarized image 32 c, it becomes possible tospecify the position of the photographic subject within the photographicimage, and its size and shape.

Variant Embodiments

It should be understood that the cameras of the embodiments describedabove may also be varied in the following ways.

(1) If the position, the size, and the shape of a photographic subjectare to be specified in a plurality of sequentially shot images that havebeen acquired by continuous shooting photography, then, apart from themethods disclosed in connection with the first through the fourthembodiments described above, the control unit 104 may also perform thefollowing processing. For example if, as shown in FIG. 16, the images 16a, 16 b, 16 c, and 16 d have been acquired in time series by continuousshooting, then first, using a technique such as that described in theembodiments above, the control unit 104 may perform photographic subjectextraction upon the image 16 a that is the first frame, thus specifyingthe position, the size, and the shape of the photographic subject inthat image 16 a. And since, in the case of continuous shooting, thechange in the photographic subject between frames is relatively small,accordingly it will be possible to enhance the processing speed byperforming processing for the subsequent frames using only the binarizedimages for which combination was performed in the step S2600, forexample the images 16 e through 16 h that are binarized from the Crplane image.

(2) In the first embodiment described above, an example was explained inwhich the control unit 104 specified the position, the size, and theshape of the photographic subject within the subject image by performingthe processing shown in FIG. 4. However, it would also be acceptable toarrange for the control unit 104 to specify at least one of theposition, the size, and the shape of the photographic subject within thesubject image, only.

(3) In the first embodiment described above, an example was explained inwhich the control unit 104 eliminated pixel islands for which theprobability of being the photographic subject was low from the subjectsfor further processing, using the secondary evaluation values #1 through#3. However, it would also be acceptable to arrange for the control unit104 to eliminate pixel islands for which the probability of being thephotographic subject was low from the subjects for further processing,using at least one of the secondary evaluation values #1 through #3.

(4) In the first embodiment described above, an example was explained inwhich, if the subject image is one in which it is possible to detect aface, then, in each of the binarized images, in the step S2400 of FIG.4, the control unit 104 specifies a first pixel island and alsospecifies, as being a second pixel island, a pixel island whose rangingtarget point is closest to that of that first pixel island and moreoverthat is close upon the screen thereto, and then specifies the position,the size, and the shape of the photographic subject within the subjectimage by combining the binarized image obtained by extracting the firstpixel island and the binarized image obtained by extracting the secondpixel island. However, it would also be acceptable to arrange for thecontrol unit 104 to specify at least one of the position, the size, andthe shape of the photographic subject, only by combining each of thewhite pixel regions with combination of a plurality of the candidatesextracted in the step S2000 whose main evaluation values are large tocombine each of the white pixel regions.

(5) In the first through the fourth embodiments described above,examples were explained in which the main evaluation values werecalculated using Equation (5) or Equation (15). However, if thephotographic subject is extracted according to a main evaluation valuethat is calculated according to the calculation equation given by thisEquation (5) from a binarized image consisting of a large number ofwhite pixels upon a background, such as for example the binarized imageof the sector 21 f of FIG. 22, then there is a possibility that trackingof the photographic subject in the second and subsequent frames maybecome unstable. Thus, in order to avoid this problem, it would also beacceptable to arrange to calculate the main evaluation values whiletaking into account the areas of the white pixels that do not correspondto any white pixel regions (pixel islands), according to the followingEquation (17):

main evaluation value=([number of white pixels in pixelisland]^(α)×number of pixels on screen)/(inertial moment of white pixelstaken around barycenter as center×number of background pixels)  (17)

It should be understood that, in Equation (17), α may be varied in therange of 1.0 to 1.5. Moreover, it is replaced by 1 if the number ofbackground pixels is 0.

(6) In the first through the fourth embodiments described above, caseswere explained in which the present invention was applied to a camera.However, the present invention can also be applied to some other type ofimage processing device that is capable of reading in images andprocessing them, for example to a personal computer or to a portableterminal or the like. In this case as well, a program that operates in asimilar manner to one or more of the embodiments described above may besupplied to the personal computer or the portable terminal as acomputer-readable computer program product, in any of various formats.

It should be understood that the present invention is not to beconsidered as being limited to any of the structures in connection withthe embodiments disclosed above, provided that its characteristicfunction is not lost. Moreover, it would also be acceptable to combinethe features of two or more of the embodiments and variant embodimentsdescribed above in various ways.

1. A photographic subject determination method, comprising: a binarization step of creating a plurality of binarized images of a subject image, based upon color information or luminance information in the subject image; an evaluation value calculation step of, for each of the plurality of binarized images, calculating an evaluation value that is used for specifying at least one of a position, a size, and a shape of a photographic subject within the subject image; and a photographic subject specification step of specifying at least one of the position, the size, and the shape of a photographic subject within the subject image, based upon the evaluation value.
 2. A photographic subject determination method, comprising: a binarization step of creating a plurality of binarized images of a subject image, based upon color difference information, luminance information, and color difference space information combined with the color difference information, in the subject image; an evaluation value calculation step of, for each of the plurality of binarized images, calculating an evaluation value that is used for specifying at least one of a position, a size, and a shape of a photographic subject within the subject image; and a photographic subject specification step of specifying at least one of the position, the size, and the shape of a photographic subject within the subject image, based upon the evaluation value.
 3. A photographic subject determination method according to claim 1, wherein: the evaluation value includes a first evaluation value that is calculated based upon an area of a white pixel region that is made up by white pixels within a binarized image, and a value that shows a state of a set of white pixels within the white pixel region.
 4. A photographic subject determination method according to claim 3, wherein: the evaluation value includes at least one of a second evaluation value that is calculated based upon an area of an enveloping rectangle that envelopes the set of the white pixels within the binarized image and an area of the set of the white pixels, a third evaluation value that is calculated based upon an aspect ratio of the enveloping rectangle, and a fourth evaluation value that is calculated based upon a size of a region that includes a face of a person.
 5. A photographic subject determination method according to claim 4, wherein: in the photographic subject specification step, from among the plurality of white pixel regions, some of the white pixel regions are eliminated based upon the second evaluation value, the third evaluation value, and the fourth evaluation value, and, from among remaining ones of the white pixel regions, the white pixel regions whose first evaluation value are large are specified as photographic subject candidates.
 6. A photographic subject determination method according to claim 3, wherein: the first evaluation value is calculated based upon at least one of the area of the white pixel regions and an inertial moment centered around a photographic subject inferred position within the subject image, an entire area of a screen, and an area of white pixels that do not correspond to the white pixel regions.
 7. A photographic subject determination method according to claim 3, wherein: the value that shows the state of the set of white pixels within the white pixel region is the area of the white pixel region and an inertial moment around a photographic subject inferred position within the image, and the photographic subject inferred position is either a position designated by a user, or a position in which a face of a photographic subject has been detected.
 8. A photographic subject determination method according to claim 7, further comprising: an inferring step of inferring a position of an upper half of a body of the photographic subject and a position of a lower half of the body of the photographic subject, based upon the position in which the face of the photographic subject has been detected.
 9. A photographic subject determination method according to claim 8, wherein: in the inferring step, a plurality of positions of the upper half of the body of the photographic subject and a plurality of positions of the lower half of the body of the photographic subject are inferred.
 10. A photographic subject determination method according to claim 8, wherein: in the photographic subject specification step, at least one of the position, the size, and the shape of the photographic subject is specified by combining the white pixel region that corresponds to the position of the upper half of the body of the photographic subject and the white pixel region that corresponds to the position of the lower half of the body of the photographic subject.
 11. A photographic subject determination method according to claim 5, wherein: in the photographic subject specification step, at least one of the position, the size, and the shape of the photographic subject is specified by combining each of white pixel regions that are produced by combining a plurality of candidates, among the photographic subject candidates, whose first evaluation values are large.
 12. A photographic subject determination method according to claim 1, wherein: in the photographic subject specification step, at least one of the position, the size, and the shape of the photographic subject is specified by combining a first white pixel region that corresponds to an inferred photographic subject position, and a second white pixel region whose ranging value is closest to the first white pixel region and that is close thereto upon a screen.
 13. A photographic subject determination method according to claim 1, wherein: hue is classified into a plurality of subdivisions; and for each subdivision of the plurality of subdivisions into which hue has been classified, a binarized image is created by binarizing the subject image according to pixels that correspond to a corresponding subdivision, and pixels that do not correspond to the corresponding subdivision, so that a plurality of binarized images that are created correspond to the plurality of subdivisions of hue.
 14. A photographic subject determination method according to claim 6, wherein: a luminance image and a color difference image are created based upon luminance information and color information in the subject image; and binarized images are created for the luminance image and the color difference image that have been created to be included in the plurality of binarized images.
 15. A photographic subject determination method according to claim 2, wherein: a luminance image, a first color difference image, and a second color difference image are created based upon luminance information and color information in the subject image; binarized images are created for the luminance image, the first color difference image, and the second color difference image that have been created to be included in the plurality of binarized images; and a color difference space achieved by a first color difference of the first color difference image and a second color difference of the second color difference image is subdivided into a plurality of color difference space subdivisions, and the plurality of binarized images are created by creating binarized images corresponding to the color difference space subdivisions, using the first color difference image and the second color difference image.
 16. A computer-readable computer program product, comprising a program that causes a computer to execute a photographic subject determination method according to claim
 1. 17. A computer-readable computer program product, comprising a program that causes a computer to execute a photographic subject determination method according to claim
 2. 18. A camera, comprising a control unit that performs a photographic subject determination method according to claim
 1. 19. A camera, comprising a control unit that performs a photographic subject determination method according to claim
 2. 